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UKRAINIAN DACTYL ALPHABET GESTURE RECOGNITION 
USING CROSS PLATFORM SOFTWARE AND CONVOLUTIONAL 
NEURAL NETWORKS 


3allpOMOHOBAaHO TeXHOJIOTIIO, pospoOueHy 3a JOMOMOroro KpociaTPopMHHX 3acoOiB, JIA MOJeIOBaHHA 2KeCTIB 
ykpaiHCbKoi WakTHWIbHOI abeTKH, aHiMallil MepexoiB MDK CTaHaMH 2KCCTOBHX OJMHHUb Ta KOMOIHyBaHHa 2%KECTIB (CIB). 
PospoOeHa TeXHOJIOTIA BIATBOPIO€ MOC OBHICTb XKECTIB 3a JOMOMOTO!0 BIPTYaJIbHOI MpOCTOpoBOi MOsesI1 pyKH Ta BH- 
KOHY€ PO3IIi3HaBaHHA JaKTHJIeM 13 BXIJHOrO MOTOKY KaMepH 3a JOMOMOTOIO HaBYeHHX Ha 3i0paHomy Hadopi 300paxKeHb 
3FOPTKOBOi HeEHpoOHHOi Mepexki. 3a JOMOMOTor0 KpOcIIaTPOpMHUX 3acoOiB TeXHOJIOTiA MOxKe OyTH 3allyLeHO!IO Ha Oara- 
TbOX IIaTPopMax 6e3 HeOOXiHOCTI MOpTyBaHHA aOo po3poOseHO!O 3aHOBO Mi, KOXKHYy WwIaTPopMy OKpemo. 

Koro 40 Bi c10Ba: KpOcriaTPOPMHICTb, MOBA 2KECTIB, MOJCJIOBAHHA JaKTHWJIeM, PO3Ili3HaBaHHA JjaKTHJIeM, 3rOpT- 
KOBI HeHpoHHi Mepexki 


The technology, which is implemented with cross platform tools, is proposed for modeling of gesture units of sign 
language, animation between states of gesture units with a combination of gestures (words). Implemented technology 
simulates sequence of gestures using virtual spatial hand model and performs recognition of dactyl items from camera 
input using trained on collected training dataset set convolutional neural network. With the cross platform means techno- 
logy achieves the ability to run on multiple platforms without re-implementing for each platform. 

Keywords: cross platform, sing language, dactyl modeling, dactyl recognition, convolutional neural networks 


Introduction 

Sign language is one of major means for 
information transition, alongside with text and 
speech. Sings can be used to define specific 
letters, words, phrases and can be processed, 
encoded and stored in a various ways. Deve- 
loping a technology for persisting and mode- 
ling signs and sign languages is a challenging 
problem due to differences in available plat- 
forms. Different platform have different ope- 
rating systems (such as mobile - iOS, Android, 
desktop - MacOS, Linux, Windows, and web - 
ChromeOS, etc), which implies different per- 
formance level and requires porting the code- 
base on each platform; some platform require 
internet connection (such as cloud computing 
technologies [1] or we operating systems) and 
others do not, etc. Presenting such a techno- 
logy for sing language is an actual problem for 
people with hearing disabilities and their rela- 
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tives, but also is important in a wider appli- 
ance, due to universality of sing language. 

Cross-platform technologies [2] provide 
a way to overcome this problem. Cross- 
platform technologies can be used instead of 
virtual-machines [3] or a set of mono-platform 
technologies. Using these technologies allows 
to develop a single codebase for different type 
of platforms, independent of CPU type, opera- 
ting system of hardware performance and to 
deploy it on all platform seamlessly. 

In this article a solution for the problem 
of sing language modeling and recognition is 
proposed based on cross-platform technolo- 
gies. The sign language modeling and recogni- 
tion performance can be flexible and adjusted, 
based on the hardware it operates or based on 
availability of internet connection. The propo- 
sed approach tunes the complexity of the 3D 
hand model (parameters such as the number of 
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polygons for rendering the hand and the ani- 
mation step of sings transition) based on the 
CPU type, amount of available memory and 
internet connection speed. The sing recogni- 
tion is also performed using cross-platform 
technologies and can be adjust the tradeoff in 
model size and performance. The sing (ges- 
ture) modeling and recognition is a part of a 
single gesture communication technology and 
this paper is a further development of author's 
previous works [4], [5]. 

Existing approaches for modeling and 
recognition of sign language and _ their 
implementtation on different platforms 

A technology for both sing language 
modeling and recognition can be considered 
as a set of separate dependent or independent 
technologies for gesture modeling and gesture 
recognition. Some of systems provide only of 
the stated technologies, typically only for a 
specific platform. American Sing Language 
Online Dictionary [6] is one of the first sys- 
tems which were assembled for gesture mode- 
ling, however it is based on a set of recored 
videos, stored in a database, and this approach 
was applied in a set of agencies [7]. However, 
due to its method of persisting gestures and no 
possible ability to modify them, this system is 
not flexible and limited only to a set of pre-re- 
corded gestures. Also, no gesture recognition 
method is provided. 

Three-dimensional hand model is an im- 
portant part of gesture language modeling. 
Two groups of hand modeling presentation are 
researched in the work [8]. In the paper, spa- 
tial approach takes into account positioning of 
the hand sign and their parametrical represen- 
tation. The temporal approach is based on the 
rules of transitions of a gesture. 

[9] develops a technology for modeling 
gesture for an input text. The technology con- 
sist of a statistical model for given text proces- 
sing and a generative algorithms for appro- 
priate hand gesture modeling, using specified 
kinematics. As a result of the work, authors 
provide ANVIL tools for annotation, DANCE 
library for sing transition and sing generator 
NOVA [10]. However, the technology is spe- 
cified to work only on Windows operating 
system and x86 CPU. Similar technology for 
modeling symbols is proposed in [11], 
however also only for a single platform. Simi- 
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lar technology for smartphones is proposed in 
[13], however the approach consists only of 
gesture recognition. 

Problem statement 

The proposed technology should consist 
of two parts, which are sign language [11], 
[12] modeling and gesture recognition mo- 
dule. Both modules should be able to run 
without codebase modification on multiple 
platforms and should be developed using 
cross-platform tools. 

Gesture recognition module should con- 
sist of a model which is able to detect and 
identify the gesture, specified by the user, 
from a camera input. Set of gestures is limited 
by the Ukrainian dactyl language, but can be 
extended further. An appropriate dataset of 
Ukrainian dactyl language should be collected 
for testing the model performance. The sing 
language modeling module should be able to 
reproduce a gesture specified by a set of para- 
meters, stored in a database, and should be 
limited by a set of Ukrainian dactyl language 
signs, but can be extended further with other 
languages. The gesture modeling module 
should also be able to reproduce gesture tran- 
sitions, meaning it can model seamlessly 
words and sentences, consisting of Ukrainian 
dactyl language signs. 

Proposed approach 

To developed a technology for Ukrai- 
nian dactyl language modeling and recogni- 
tion, which can run on multiple platforms, 
without changing the codebase, an approach 
based on cross-platform tools is proposed. 
Gesture modeling module should consist of a 
virtual three dimensional hand model and a 
user interface, which should provide the user 
with ability to specify a symbol or a set of 
symbols, which then will be transitioned as a 
sequence of gestures. To implement both hand 
model and user interface, a cross-platform 
framework Unity3D [14] was used. Compa- 
ring to other 3D engines, it provides a unified 
development process for all available plat- 
forms (mobile, desktop and web) and provides 
a seamless way to deploy the application on 
all of them without changing the codebase. To 
develop a gesture recognition module, a cross- 
platform framework Tensorflow [15] is propo- 
sed. This approach based on cross-platform 
framework for machine learning allows to 
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developed and train a gesture recognition mo- 
del once, and then deploy it on multiple plat- 
forms (mobile, desktop and web) without any 
modifications to the model or the code for trai- 
ning. As a model architecture, the MobileNet 
architecture is considered, enhanced with 3D 
convolutions, to take into account temporal in- 
formation from a sequence of input frames 
from the camera. Altogether, the proposed 
technology novelty is that it's a unified cross- 
platform technology for Ukrainian dacty] lan- 
guage modeling and recognition, with im- 
proved MobileNet architecture for improved 
recognition of the Ukrainian dactyl alphabet. 

Infologic model 

The technology architecture diagram 
(Figure 1) demonstrates the interaction of main 
components of the proposed system. The gestu- 
res for gesture modeling are stored in a speci- 
fied format (YAML [16]) in a database, and are 
exploited by the gesture modeling engine for 
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A Gesture 
Gesture Modeling module Specifications 


Model representation 
of the hand (Unity3d) 


Hand model 


setting a configuration of a spatial three- 
dimensional hand model using specified para- 
meters for the gesture from a corresponding 
database entry. The gesture modeling module 
operates over the gesture database and is a part 
of the application, which consists of gesture 
modeling and UI components, both of them 
being developed with Unity3D framework, 
using C# programing language. The virtual 
hand model is specified by a skeleton and a set 
of parameters and their limitations for each ske- 
leton joint. The gesture recognition module is 
implemented with Tensorflow framework, 
using Python programing language. The ges- 
ture recognition module run independently of 
gesture modeling module and uses gestures 
database. Main components of the gesture re- 
cognition module is the model which performs 
gesture recognition, and the wrapper which 
converts camera input into appropriate format 
for the model. 


Gesture 
specifications 


Gesture storage (YAML) Gesture Recognition module 


Fig. 1. Infologic model of cross platform gesture communication technology. 


Gesture modeling 

The three-dimensional hand model ske- 
leton was implemented with real human hand 
anatomy in mind. Skeleton model consists of: 
8 bones in wrist, 3 bones in the thumb and 1 
metacarpus and 3 phalanges in each of the 
other fingers. Each joint of each pair of bones 
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has it's own type of connection and it's own 
parameters for setting this joint, it's own 
degree of freedom and it's limitations. Overall, 
the hand model is represented with a skeleton 
which consists of 27 bones and has 25 degrees 
of mobility. The thumb has 5 degrees of free- 
dom, middle and index fingers have four deg- 
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rees of freedom, four degrees of freedom are 
located in the metacarpal-carpal joint to the 
little finger and thumb to enable movement of 
the palm. 

Unity3D framework was used for im- 
plementing the three dimensional hand model, 
since developing your own cross-platform ren- 
dering engine is a non-trivial task. Unity3D 
was selected due to friendly user interface, 
ability to implement through it's means both 
the scene and user interface. Over the hand 
skeleton, a realistic hand model was develop- 
ped, rendered with more than 70,000 polygons 
(Figure 2), Unity3D framework is able to han- 
dle such model with satisfactory performance. 


Fig. 2. Gesture modeling under 
iOS platform. 


Gesture recognition 

Gesture recognition, as a part of cross- 
platform technology for Ukrainian dactyl lan- 
guage modeling and recognition, should be 
implemented using cross-platform tools. Ges- 
ture recognition approach depend on the type 
of input information they work with. In case 
of 3D model bases algorithms or skeletal-ba- 
sed algorithms, the approach can use volu- 
metric or skeletal model, or a combination of 
them. Although, these approaches tend to be 
computationally expensive and require addi- 
tional hardware from user. Other type of ap- 
proaches, appearance-based models derive pa- 
rameters directly from the image or a se- 
quence of images (in case video is used as an 
input). As a next step some pattern mining 
technique or machine learning approach is 
used to train a recognition model. Due to no 
need in additional hardware apart from a 
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simple webcamera, these type of approaches 
were selected for the cross-platform techno- 
logy. Some approaches, for example, Ong et 
al. [17] proposes Sequential Pattern Mining in 
order to detect signs based on the tree 
structures. 

Convolutional Neural Networks (CNN) 
is a class of deep neural networks which are 
regularized versions of multilayer perceptrons, 
most commonly applied to analyzing images 
and videos. CNNs are especially good at ana- 
lyzing images due to ability to take into 
account locality reference of the data in the 
image (typically nearby samples at some input 
data are not related, which is not true in case 
of an image). Therefore, CNN show state-of- 
the-art results in image classification and re- 
cognition tasks [18], [19]. Another benefit of 
the convolutional neural networks is no need 
in hand-crafted features, unlike conventional 
pattern matching algorithms. The process of 
training takes the input data and finds all the 
features needed for recognition and _ stores 
them as weights of the model. CNNs are ro- 
bust at the task of classification or recognition 
of the object on an image, independent of in- 
put image scale, lightning conditions, occlusi- 
ons, noise, etc. Although training such a mo- 
del requires a sufficient dataset. Typically ar- 
chitecture of the CNN consists of a set of con- 
volutional, pooling and ReLU layers. Tensor- 
flow framework provides a cross-platform and 
performance-efficient implementation of con- 
volutional neural networks. 


Input Operator t c n|s 
2242 x 3 conv2d - 32 2 
112? x 32 bottleneck | 1 16 


112? x 16 bottleneck | 6 | 24 
56? x 24 bottleneck | 6 | 32 
282 x 32 bottleneck | 6 | 64 
14? x 64 bottleneck | 6 | 96 
14? x 96 bottleneck | 6 | 160 
72 x 160 bottleneck 6 | 320 


NNN 


no 


DW WD ee 
+ ome 


7? x 320 conv2d 1x1 | - | 1280 
7? x 1280 | avgpool 7x7 | - - 
1x 1x 1280 | conv2d 1x1 | - k 


Fig. 3. Architecture of MobileNetv2. 


Gesture recognition experiment 

Training a CNN model requires an ap- 
propriate dataset. The data should not be bia- 
sed, therefore for the task of Ukrainian dactyl 
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alphabet gestures recognition the dataset 
should be diverse. Since the temporal informa- 
tion also provides useful information about 
gestures transitions, each sample of the ges- 
tures should exist as a sequence of frames or a 
short video, which is also increasing the data- 
set. For the gesture recognition experiment a 
dataset of 50,000 images was collected, with 
50 different hands recorded, with almost 1,000 
images per each person. Distribution of male 
hands was 55% and female 45%. 

The dataset was collected using differ- 
rent surroundings. The background was chan- 
ging for each person after each 50 frames re- 
corded. Different lighting conditions were 
setup during recording (20%/30%/50% in low, 
medium, bright light conditions correspon- 
dingly). 20% of the dataset was distorted with 
noise and blur with gaussian noise and gaus- 
sian blurring. 

MobileNetV2 [20] architecture (Figure 3) 
is anew mobile architecture, development of the 
MobileNet model. MobileNetV2 extends its 
predecessor with 2 main ideas. Residual blocks 
connect the beginning and end of a convoluteo- 
nal block with a skip connection. By adding 
these two states the network has the opportuni- 
ty of accessing earlier activations that weren’t 
modified in the convolutional block. This 
approach turned out to be essential in order to 


TotalLoss 


build networks of great depth. On the other 
hand, MobileNetV2 follows a narrow->wide- 
>narrow approach. The first step widens the 
network using a 1x1 convolution because the 
following 3x3 depthwise convolution already 
greatly reduces the number of parameters. 
Afterwards another 1x1 convolution squeezes 
the network in order to match the initial num- 
ber of channels. In the table (Figure 3) it is how 
the bottleneck blocks are arranged. t stands for 
expansion rate of the channels. A factor of 6 
opposed to the 4 in our example. c represents 
the number of input channels and n how often 
the block is repeated. Lastly s tells whether the 
first repetition of a block used a stride of 2 for 
the downsampling process. This is a common 
assembly of convolutional blocks. 

Process of training MobileNetv2 net- 
work for gesture detection takes ~ 200.000 
iterations. Figure 4 shows curves of how the 
neural network is optimized. Such a training 
allowed to achieve accuracy of 98% on testing 
dataset, of additional size of 15,000 images, 
with 30 different hands recorded, with almost 
500 images per each person. 

Figure 5 demonstrates user interface of 
gesture recognition module. Letter B of Ukrai- 
nian dactyl alphabet is detected and shown 
with a bounding box. 


regularization_loss_1 


Fig. 4. Loss and regularization loss curves of MobileNet training process. 


Application of cross platform tools 

The proposed technology is imple- 
mented using cross platform tools. The gesture 
modeling and recognition modules are both 
developed using cross platform frameworks, 
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which operate over a databased with gestures 
in a unified format (YAML). To overcome the 
problem of running the technology with satis- 
fying performance on multiple platform, set- 
tings for performance adjustment were deve- 
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loped. Based on the platform and its initial 
hardware analysis, the dimensional hand mo- 
del is adjusted. Number of polygons can be 
lowered, and step of performance can be 
decreased. 

However, there are no specific hardware 
requirements for the technology to run, as the 
used cross-platform frameworks (Tensorflow, 
Unity3D) don't require. 

If the available hardware does not meet 
the minimum requirements of information tech- 
nology, the user is given the recommendation 
to choose “online” mode, in which the calcula- 
tion is not performed on hardware. 


Fig. 5. Example of recognition UI. 


Conclusions 

The proposed technology consists of two 
main modules: gesture modeling and gesture 
recognition modules, which use the database 
with gestures specifications stored in YAML 
format in a PostgreSQL [21] database. 

The proposed technology implements 
gesture modeling and gesture recognition for 
Ukrainian dactyl alphabet gestures. A dataset of 
50.000 images was collected using diverse con- 
ditions and different persons hands. Gesture 
modeling was implemented using Unity3D 
framework, which is cross-platform and shows 
satisfying performance on different platforms 
(mobile, web and desktop) while rendering a 
realistic three dimensional hand model. Num- 
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ber of polygons and animation step of gesture 
transitions can be adjusted for the sake of per- 
formance. Gesture recognition module was im- 
plemented using Tensorflow framework, which 
provides ability to deploy its model on different 
platforms without any codebase modifications. 
As a model for gesture recognition, 
MobileNetv2 architecture was chosen, as a mo- 
del with best trade-off of size and accuracy, 
especially on low performance platforms (such 
as mobile and web). The model was trained on 
the collected Ukrainian dactyl language dataset. 

The proposed gesture communication 
technology can be further augmented with 
other gestures and languages, and with other 
cross-platform modules. 
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PE3SIOME 


C. C. Konapariok 

PosnisHaBaHHA >%KecTIB yKpalHCbKol 
aKTHJIbHO! adeTKH 3a JOMOMOrOIO Kpoc- 
I1aTPOPMHHX TexHOJOTi Ta 3rOpTKOBHX 
HeMpOHHHX MepexK 

Moza 2KecTIB € OJHUM 13 OCHOBHHX 3aCc0- 
6iB Tepeyaui indopMaliii, Mopay i3 TeKCTOM 1 
MOBOIO. AK IpaBHJIO, Y KOXKHOI KpalHH € CBOA 
plqHa MoBa >2KeCcTIB, IIpoTe HalleBHO HeBiOMO, 
CKUIbKH MOB 2KeCTIB ICHY€ B YCbOMY CBITI. 
YxpaiHcbKa MoBa %KecTIB Ta YKpalHCbKHi aJi- 
(paBiT TaKTHJIeM € OHM 13 HaliMolmmMpeHiliHx 
3acoOiB CIIKyBaHHa B YKpaiHi MicsIa TeKCTO- 
BOFO Ta pO3MOBHOTPO CIUIKYBaHHA. 

HayaHua TeXHOJIOrii BUBUCHHA 2KeCTIB 
(3HaKiB, JaKTHJIeM) yKpaiHCbKOi MOBH JIA Ta- 
KOI CIIWIbBHOTH € AKTYAJIbBHOIO MpoOseMoro Ta 
CKJIaJHHM 3aB{aHHAM. 

Jia BUpPILIeHHA 3aBaHHA MOesIKOBaH- 
HA MOBH 2KECTIB Ta BAKOH@HHA aHiMallil 2Kec- 
TOBHX CIPyKTyp 3a JOMOMOTOr!O IIpocTopoBoi 
BIPTYaIbHO! MOJ{eJIi PYKH, IpomoHyeTECA 
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KpocraTdopMHa TeXHOJIOrA, 3aCHOBaHa Ha 
KpocriiaTpopMuiit O1iOmioteui Unity3D. Kpoc- 
Ti1arpopMua Oidsioreka Unity3D Tako Bu- 
KOPHCTOBY€TBCA JIA inTepdelicy Kopucty- 
Bada, TEXHOJIO“A peasli30BaHa 3a JOMOMOTOrO 
MOB IIporpaMyBaHHa C#. 3allpomoHoBaHi 
iHCTPyMeHTH MOXKYTb BUpPIWINTH mpodsemy 
3allyCKy TeXHOJIOri Ha JCKiJIbKOX 1CHYIOUHX 
TuiaTdopMax. Hosu3Ha 3all1pomoHoBaHoi Tex- 
HOMOrii MoMAra€ B TOMY, UO BOHa € Kpocriat- 
(popMHo}0 Ta Ma€ HaCTpOIOBaHH PiBeHb MoIIi- 
TOHIB JJIA TPHBUMIPHO! MOJeJ pyKU Ta KpOK- 
aHIMallii [VLA TepexOIB *KeCTIB. 

Mogemb pyku, BOy,OBaHa B MOJLYIb 
MOJ[€JIKOBAHHA 2KeCTIB, Mae 27 KICTOK, KO2KHA 
KicTKa 3'€{HaHa 3 iHIWOIO Yepe3 pi3Hi THIM 
cyrmo00iB. Y paMkax TexHosorii MOj{esIIOBaH- 
HA peasii30BaHO TPHBUMIPHY MOJ{eJIb PyKH Ta 
peami30BaHO aHiMallii MepexoJiB MDK 2KecTa- 
MH (Mopdemamn). JjaHa TexHOJIOriA 3aTHa 
BIATBOPHTH peaslicTH4Hy MOj[eJIb PyKH, WO 
cklaqjaeTbcea 3 WOHay 70000 momironis. 

Mogy.sli po3ri3HaBaHHA 2KecTIB, po3po0- 
JIeHi 3a OMOMOro!IO KpocraTPOpMHUX 1H- 
CIPyMeHTIB (3aCHOBaHi Ha Python, C ++), Mo- 
%KYTb OyTH BOyAOBaHI B 1HopMalliitHy TexHO- 
worit0. KOHBOSIOWIMHI HeiMpOHHi MepexKi T10- 
Ka3aid HaHi pesyIbTaTH B 3ayaqax 3 po3- 
Il3HaBaHHA 300parxKeHb Ta 2%KecTiB. Jia eKcrie- 
puMenty Oys 3i0paHHit Hadip WaHHx 3 WakTH- 
JleMaMH =ykKpaiHCbKOi MoBH. KoxkKeH %xeCT 
ckagaetaca 3 1000 3pa3koBHx 300parxKeHb. 50 
PI3HHX JHOTeH WOKa3yBasI X%KeCTH: 3 POsIOyI- 
mom 70% YornoBianx Ta 30% %*iHOUMX PpyK. 
ByJIM BUKOpHCTaHI pi3Hi YMOBH OCBITIIeHHA (3 
po3rmoiI0M 20% 300paxKeHb y ToraHux, 30% 
y mocepeguix ta 50% y xopomwmMx ymMoBax 
ocBiTIeHHA), 10% 300paxKeHb OysIM CIOTBO- 
PpeHi LYMOM Ta po3MHTTAM. 

Apxitextypa MobileNet Oyma Bukopuc- 
TaHa AK OCHOBa JIA apxirektypu CNN, i HaB- 
yaHHa TpuBalo ~ 200 000 irepamiii, m0 cTa- 
HOBHTb WpHOmm3HO 10 emox, 1 WocarHyTo 
~98% TOUHOCTI Ha TeCTyBaJIbHOMy HaOopi 
7aHux. 
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